Manual rollback after grace period by pchila · Pull Request #9643 · elastic/elastic-agent

pchila · 2025-08-29T09:49:17Z

What does this PR do?

This PR builds upon #8767 by allowing to manually rollback an elastic agent even after the grace period ends.
This PR uses a registry of elastic-agent available rollbacks introduced with PR #10344.

Having a registry of agent versions available for rollback allows for detection of possible manual rollback targets. The same available rollbacks are written in .update-marker file so that the watcher will preserve them at the end of the grace period, still allowing for a rollback after the watcher exits.

Installs available for rollbacks are assigned a TTL, so they can be cleaned up after a given time (governed by the agent.upgrade.rollback.window setting).

With this PR, cleanup and normalization of agent installs happens only at startup.
In a follow-up PR:

the agent will be able to schedule a cleanup to run without needing a restart
old agent installs available for rollback will be cleaned up when initiating a new upgrade operation (in order to save disk space)

Why is it important?

This PR allows to manually rollback an elastic-agent upgrade within a configurable window that extends beyond the grace period (the period during which an automatic rollback may be triggered by the upgraded agent misbehaving).

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~[ ] I have added an entry in ./changelog/fragments using the changelog tool~~
I have added an integration test or an E2E test

Disruptive User Impact

No impact for users as the feature is deactivated when using the default value of agent.upgrade.rollback.window: 0

How to test this PR locally

Package elastic agent twice from this PR:

SNAPSHOT=true EXTERNAL=true PACKAGES=tar.gz  PLATFORMS="linux/amd64" mage -v package

AGENT_PACKAGE_VERSION="9.3.0+build20251022000000" BEAT_VERSION="9.3.0-SNAPSHOT" EXTERNAL=true PACKAGES=tar.gz  PLATFORMS="linux/amd64" mage -v package

Install the version 9.3.0-SNAPSHOT as usual
Set a rollback window duration > 0, for example 10 minutes and a shorter grace period and check_interval for watcher:

agent.upgrade:
  watcher:
      grace_period: 1m
      error_check.interval: 5s
  rollback:
      window: 10m

Trigger an update to the other package (saved on disk):

elastic-agent upgrade --skip-verify --source-uri=file:///vagrant/build/distributions 9.3.0+build20251022000000

Wait for the new agent to come online and check the upgrade details for UPG_WATCHING state. After the grace period of 1 minute, upgrade details should disappear and the watcher should exit.
Verify that the data/elastic-agent-9.3.0-SNAPSHOT-<hash> directory is still present after watcher exited.
Manually rollback to the previous version:

elastic-agent upgrade --rollback 9.3.0-SNAPSHOT

Check the output of elastic-agent status and verify that the agent restarted with version 9.2.0-SNAPSHOT and that the directory data/elastic-agent-9.3.0+build20251022000000-<hash> contains only logs.

Related issues

Requires Introduce install TTL markers #10344

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

mergify · 2025-08-29T09:49:53Z

⚠️ The sha of the head commit of this PR conflicts with #8767. Mergify cannot evaluate rules on this PR. ⚠️

mergify · 2025-08-29T09:50:37Z

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

mergify · 2025-09-09T01:17:17Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

mergify · 2025-09-23T07:30:53Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

mergify · 2025-09-25T12:27:37Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

mergify · 2025-09-26T15:34:57Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

elastic-sonarqube · 2025-09-29T20:53:44Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

mergify · 2025-10-01T18:06:12Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

elasticmachine · 2025-10-03T12:32:19Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

internal/pkg/agent/application/upgrade/manual_rollback.go

internal/pkg/agent/application/upgrade/rollback.go

internal/pkg/agent/cmd/watch.go

testing/integration/ess/upgrade_rollback_test.go

internal/pkg/agent/cmd/watch.go

internal/pkg/agent/application/application.go

pchila · 2025-10-28T18:31:27Z

buildkite test this

cmacknz · 2025-10-28T19:40:39Z

Code looks good to me, I went through the manual test steps and it worked with the following observations:

I don't see rollbacks available reported from the status command, just the watching state:

❯ sudo elastic-development-agent status
┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
├─ elastic-agent
│  └─ status: (HEALTHY) Running
└─ upgrade_details
   ├─ target_version: 9.3.0+build20251022000000
   ├─ state: UPG_WATCHING
   └─ metadata

The agent status appears that it will indefinitely contain the rolled back status, this is fine since that's what happened. Someone may ask us to change this later I think it will permanently highlight the agent in the Fleet UI with the rolled back status which is less useful if it was manual and not automatic.

❯ sudo elastic-development-agent status
┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
├─ elastic-agent
│  └─ status: (HEALTHY) Running
└─ upgrade_details
   ├─ target_version: 9.3.0+build20251022000000
   ├─ state: UPG_ROLLBACK
   └─ metadata
      └─ reason: manual rollback requested to version 9.3.0-SNAPSHOT

cmacknz · 2025-10-28T19:42:14Z

@ycombinator still has open comments to resolve, @ycombinator can you confirm these are addressed and approve? This LGTM but I want to avoid approving when there are open comments from someone else remaining.

ycombinator

@pchila Thanks for addressing my comments. I've resolved most of them, just a couple where I'd like to see a bit more in the inline comments explaining the "why" + one other question about some code. After that, this LGTM!

elasticmachine · 2025-10-30T15:02:31Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: f1568f6

Failed CI Steps

fips:x86_64:sudo-false:fleet

History

💔 Build #29651 failed 52ab103
💚 Build #29535 succeeded dcf0ede
💛 Build #29101 was flaky db96ca9
💛 Build #28919 was flaky 6f4272b
💛 Build #28851 was flaky 53cdd8a

cc @pchila

ycombinator

LGTM!

* Allow for multiple directories to be specified during cleanup * refactor manual rollback function and tests on a separate file * Split manual rollback between watching and non-watching cases * Implement manual rollback from list of agent installs * fix lint errors * Normalize install descriptors at startup * Add integration test for manual rollback after grace period * fix linter errors * Set commit hash in TTLMarker when preparing available rollbacks * Pass versionedHomesToKeep to installModifier.Cleanup() * change check for running TTL marker normalization at startup * remove references to install registry * implement code review feedback * fixup! implement code review feedback

mergify bot assigned pchila Aug 29, 2025

mergify bot mentioned this pull request Aug 29, 2025

Elastic Agent upgrade: lock-free manual rollback #8767

Merged

4 tasks

pchila force-pushed the rollback-watcher-cleanup branch 2 times, most recently from 33afefd to 1c65022 Compare September 17, 2025 12:06

pchila force-pushed the rollback-watcher-cleanup branch from 74ae1d1 to c51fb1b Compare September 23, 2025 12:37

pchila force-pushed the rollback-watcher-cleanup branch from 5199c2b to 38b9b96 Compare September 25, 2025 13:31

pchila force-pushed the rollback-watcher-cleanup branch from 4c89a3b to 411205b Compare September 29, 2025 16:50

pchila force-pushed the rollback-watcher-cleanup branch from 29c1a6b to cb90912 Compare October 2, 2025 06:33

pchila added enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-skip labels Oct 2, 2025

pchila force-pushed the rollback-watcher-cleanup branch from e18f176 to 2dbed3f Compare October 3, 2025 08:17

pchila added the skip-changelog label Oct 3, 2025

pchila marked this pull request as ready for review October 3, 2025 12:32

pchila requested a review from a team as a code owner October 3, 2025 12:32

pchila requested review from straistaru and ycombinator October 3, 2025 12:32